UBELT
Test Description: University of Bath English Language Testing for Medical Professionals (UBELT) ' ' Test Purpose The purpose of the test is to: · assess the English level of non-native speaker candidates who are applying for professional positions within the medical field in the UK · distinguish between candidates whose level of English is good enough to communicate sufficiently well in the native speaker professional context and those whose level is not · identify areas of weakness and allow recommendations to be made about the circumstances (if any) under which candidates should be re-considered in cases where the candidate’s language level is assessed to be below the required level. Broad test creation guidelines · All tests are made with the following in mind: selected candidates will work in an English-speaking/ native speaker environment. Selected candidates will have to deal with the British public as well as with colleagues. · The test is designed to identify candidates who are at upper intermediate and advanced levels, corresponding to levels B2 through C2 of the Common European Framework or levels 2.5 and above of UBELT. · Different professions may choose to have different ‘pass’ levels, corresponding to the different needs of professionals in these areas Test Takers ''' Test takers have been identified in terms of the following: i. All have higher education qualifications. ii. All are over 20 years of age. iii. Some are final year students in a medical profession course (such courses include a component of work experience/ placement), or they may be very recent graduates; in both cases, their experience is not broad. iv. Most are medical professionals with work experience in their own country. v. All have a shared knowledge of certain areas, usual practical routines, and information common to their profession. vi. Some have not learned English language in a formal teaching environment. vii. Many are without explicit training in language test taking. viii. Many have no previous experience of language tests. ix. Many may well feel apprehensive or nervous in the test taking environment due to the high stakes involved. x. The vast majority who attempt UBELT are at an intermediate and above language level, corresponding to level B1 and higher of the Common European Framework. xi. Many have been pre-selected by third parties according to their English level and clinical knowledge xii. Most candidates are from European language groups; some are from non-European language backgrounds, but are living in a European country (thus, they may be able to communicate in 3 or more languages). xiii. Most are living in a country outside the UK, but some are already living and /or working in the UK. xiv. All want to live and work in the UK in their chosen profession. Feedback of assessment results to test takers i. The certificate shows their overall UBELT score and includes language level descriptions for the UBELT score range. ii. Results are validated by e-mail request. iii. Comments and feedback are available on the test taker’s performance. iv. If test takers have not met their profession’s required score, they can contact the UBELT office to ask for advice on areas of weakness, how to improve, and how to apply for a re-sit. Such consultation is kept brief. UBELT does not offer any language or test preparation courses. '''Test Users Test users have been identified as the test takers, recruitment agencies, health profession employers- in both the private and public sectors - and professional boards or governing bodies. Feedback to the test users is via- *direct contact between the examiner and the agency and/or employers in a feedback session at the end of the test day. *general feedback to interested parties: e-mail and telephone conversations. *annual reports are available to all interested parties, on request. *validation studies are available to all interested parties, on request. Feedback from the test users to UBELT is via- *direct contact between the examiner and the agency and/or employers in a feedback session at the end of the test day. *direct contact with the UBELT staff: face-to-face, e-mail and telephone. *regular feedback surveys conducted by UBELT. These contribute to UBELT’s on-going feedback loop, and specifically contribute to validity studies. Target Language Use (TLU) The communicative context for candidates will be daily life and professional, consequently: *all test tasks, and content is based on professional requirements and daily life realities. *test tasks require language knowledge and skills, as well as general professional topic knowledge. · test tasks are related to the broader topics of the profession as they seek to minimise as much as possible any advantage that someone with a specialisation in the profession might have. *decisions as to what is required in real-life are based on consultation with employers, recruitment agencies, British professionals working in the same area, and foreign language professionals. *detailed reports containing information from professional informants are kept and referred to regularly by test developers. *test creators select from the suggested typical topics and tasks to create test tasks for each component. Component Descriptions The test includes 4 components, each relating to one of the four skills areas often used in language assessment. It is recognised, however, that each component includes to some extent the use other skills, and all require a level of general communicative competence. Constructs and Validity of UBELT tests The purpose of the test is to assess overall communicative competence rather than to assess correct usage and knowledge about the English language, thus the constructs are broad rather than highly specific. The test prioritises competent linguistic communication over strict notions of accuracy. While accuracy is an important factor in effective communication, UBELT is not an assessment of individual language items in isolation (e.g. discrete grammar questions); instead it focuses on the extent of the candidate’s ability to use English effectively and appropriately in the given context. A continual feedback loop involving medical professionals, candidates and examiners is encouraged by UBELT, and forms an important part of its continued test development. Content Validity ' UBELT’s content is related to the information, topics, and types of communication required for the profession in question. Since the real professional work situation will be in the U.K., the test is based on authentic native-speaker language, however, great care is taken to maintain an adequate level of standard language, style and content to prevent discriminate against those without specialist or area knowledge. Thus, professional content is related to general information and issues, and performance assessment is related to language usage, communication skills and strategies instead of ‘clinically correct’ answers. Content decisions are based on information from different sources. This process can be seen by taking the pharmacy test as an example. Information collection from professional informants to determine appropriate content and contexts. *This was done in conjunction with two experienced pharmacists, and recruiters. · A pilot test was conducted with a small number of pharmacists from the University of Bath. Their feedback and test results were used to develop Version One of the test. · A small group of recruiters (all pharmacists themselves) took the formal written tests and their feedback was collected in person, on the phone and through questionnaires (November 2004 and January 2005). Their feedback was favourable. · A discussion of the test needs and a review of the first version took place with the Director of Studies, Advanced Programmes in Pharmaceutical Practice & Therapeutics at the University of Bath. His opinion of the test was very favourable. His information and input was recorded and has fed directly into test development. · 2 interviews were conducted with non-native speakers who work in Britain as pharmacists. Their information and input was recorded and has fed directly into test development. One of these also participated in a pilot reading of a new version reading test. · Genre information and topic collection was done by a team of test developers via a survey of pharmacy websites, products and popular journals. Test developers continue to use these sources as guides for up-to-date topics and information. English language and communication information This is the development team’s own area of expertise. They are assisted in this by an independent test development consultant with many years of experience in testing. '''Authenticity and Face Validity ' Authenticity is an important consideration in communicative language testing. UBELT seeks to do this in a number of ways. The actual processes and strategies candidates are required to employ to perform in the test are “authentic” in that they simulate as much as possible typical communication in normal professional contexts. For example, in the speaking test, unlike many other language tests, the examiner engages to some degree in the discussion with the candidate to simulate natural communication. Also, when deciding on the speaking score UBELT considers the level of interactive ease of the candidate and their comprehension of the English as spoken by the interviewer, as well as taking into account the language produced by the candidate. All of these facets of communication are essential in real-life situations. The writing test uses straightforward tasks, which assess language and communication skills rather than formulaic formats and knowledge of strict academic writing conventions. The reading and listening tests use texts and situations which are drawn from some of the topics and exchanges the professional is likely to encounter in the UK, including some current colloquial usage. Standard question types are used. From UBELT’s feedback studies, it has been made clear that UBELT is a user-friendly test in its length and format. A further significant point is that it has a high “face validity” factor, i.e. it includes professionally relevant tasks and topics and so can appear more credible as a test to candidates. As a result, candidates are able to approach the test with greater confidence. =Assessment & validity = Criteria and Descriptors Assessment criteria and descriptions of language competency have been developed in the following ways: i. according to current language test practices ii. according to guidelines in the Common European Framework of References for Languages iii. after careful scrutiny with our external test consultant iv. from empirical data gathered from examiner experience, user feedback, as well as moderation of written tests and recordings of speaking tests. The assessment descriptors and reliability procedures for the Dental and Pharmacy tests are of such a nature that they can be applied directly to the other professional tests. Scoring Method Authenticity is also enhanced by the UBELT scoring method which focuses on the actual professional performance needed for the job. This is fundamental to our test design and delivery. For example, while UBELT includes all four language skills in its assessment of a candidate, the overall score is weighted in favour of the skills which are most significant for working in the U.K. The most important language skills for each profession are signalled through consultation with professional informants- practising professionals, employers, recruiters and officials of governing bodies. For example, in both Dentistry and Pharmacy, speaking and listening have been prioritised and given weight accordingly. Concurrent Validity Concurrent validity involves comparing candidates’ scores obtained by taking UBELT with other known measures of their language ability. In the design and piloting stages, information is collected about the volunteers to investigate the extent to which UBELT scores concur with their other language assessments, both formal and informal (e.g. from their teachers, other language tests, informal conversations). In the live test stage, UBELT examiners note any comments made by clinical recruiters about candidates’ performance in clinical interviews and report these back to the UBELT project director. Predictive Validity While it must be accepted that no language test can ever replicate the complexity of the real communicative situation, a fundamental premise of all language tests is that the language test samples the candidate’s language ability, measures it, and the results can provide a clearer picture of candidates’ language level. From this the language tester and test user seek to make predictions about how well the candidate is likely to perform in future situations (predictive validity). In the UBELT test, such an interpretation and prediction hinges on the candidate’s level of ease, fluency, and effectiveness in using English. The data to support the UBELT’s claims for a high level of predictive validity derives from tracking candidates’ subsequent performance in the professional situation. Information from language level assessment surveys after the candidates have begun work in the U.K is gathered regularly, and requires support from the candidates as well as agencies or employers. ' ' '''Reliability =Reliability in Test Design= i. All UBELT tests are designed according to a blueprint. ii. All UBELT test creators are trained and work in collaboration with other creators and a supervisor. iii. Versions of reading and listening tests are carefully designed, scrutinised and piloted according to standard language test procedures. Tests are piloted first with native speaker teachers, after which the test is re- evaluated and re-written as necessary. Tests are then piloted with mixed groups of students, some of whom have known other English assessment scores so that comparisons can be made with the test results. These groups include volunteers from several science areas, including pharmacy and chemistry. Versions of tests are calibrated against each other. The data from tests used in live situations, is collected for statistical analysis. iv. There is on-going feedback and monitoring of components. Examiners report back on test events; they include information on any questions, comments, observations, or variations encountered. All components are monitored and statistics are collected regularly. Examiner reliability Of crucial importance is the reliability and consistency of the examiners’ delivery and judgements. UBELT examiners have a high level of language teaching and assessment experience. Within UBELT, examiners undergo careful initial training, regular standardisation, self-reflection, moderation, and monitoring. ' ' References Category:Language proficiency measures